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ABSTRACT 

Lower than expected high-stakes examination scores were being observed in a first-year 
general psychology class. This research sought an alternate approach that would assist 
students in preparing for high-stakes examinations. The purpose of this study was to measure 
the effectiveness of an alternate teaching approach based on the testing effect to address low 
high-stakes examination scores. This was accomplished through the introduction of an online 
quizzing application that utilized a game show-like user interface called Kahoot™. The results 
showed a significant difference in high-stakes examination scores for students who utilized 
Kahoot™ versus students who did not. It can be suggested that pedagogical tools like 
Kahoot™ have the potential to enhance and improve high-stakes examination scores at the 
college and university level. Students that used Kahoot™ felt positive about their 
experience. The results of this study also suggest that creating a fun and engaging 
environment also supports improved academic performance. 

Keywords: Educational technology, self-regulation, formative assessment, gamification, 
testing effect. 

INTRODUCTION 

Asa professor in higher education fora number of years now, I have seen a common perplexing 
pattern of student behavior. Students who reportedly study diligently and are engaged in class 
are not always the ones that excel in summative course examinations. Two thoughts 
immediately come to mind. Either I am being deceived by the perceived practices of my 
students or how my students study and prepare for the examinations needs to improve. I 
decided to focus on the latter and my inquiry led me to a memory retrieval practice called the 
testing effect. Will the practice of the testing effect in a higher education setting influence 
high-stakes examination scores? Finding an answer to that question was the motivation 
behind this study. 
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The objective of this pilot quasi-experimental action research study was to measure the 
effectiveness of an alternate teaching approach based on the theory of the testing effect to 
address low high-stakes examination scores. The sample groups are from an undergraduate 
level psychology course in a four-year university located in the south Pacific. The intervention 
used to implement the testing effect utilized an online quizzing application that utilized a game 
show-like user interface called Kahoot™. This research assessed the effectiveness of this 
approach by comparing high-stakes examination scores and also by surveying the sample 
groups for their feedback on their experiences through this study. 

LITERATURE REVIEW 

The testing effect is a robust and reliable phenomenon that demonstrates that taking an initial 
test improves performance on subsequent tests (Chan & McDermott, 2007). The testing effect 
(or retrieval practice effect) has traditionally been measured in the laboratory setting, 
however, evidence of the testing effect has recently been shown in real-world settings 
(Agarwal, Bain, & Chamberlain, 2012). Low-stake quizzes have shown improved summative 
exam scores over the course of a semester relative to not being quizzed (Roediger, Agarwal, 
McDaniel, & McDermott, 2011a). Roediger et al. (2011a) found that sixth grade social studies 
students performed better on material they had been quizzed on throughout the semester, 
performed better than students who were tasked to re-read material, and also performed 
better when tasked to self-quiz themselves outside of the classroom. The time-delay between 
exposure to material, and chapter and final exams, were typically one to two months, which 
was much longer than previous studies conducted in a laboratory setting. This showed 
evidence of the long-term benefits from retrieval practice learning. The testing effect was also 
demonstrated in eighth grade science students who underwent a longer time-delay of five to 
eight months between the teacher's lessons and chapter and final exams (McDaniel, Agarwal, 
Huesler, McDermott, & Roediger, 2011). 

McDaniel, Anderson, Derbish, and Morisette (2007) also demonstrated the testing effect 
among college students. Students took weekly quizzes followed by unit tests and a final 
cumulative exam. Quizzes were both short answer and multiple-choice format with the unit 
tests and exam in a multiple-choice format. Improved performance was seen with quizzing 
versus additional reading. 

The testing effect has also been shown in novel situations (McDaniel, Thomas, Agarwal, 
McDermott, & Roediger, 2011). Experiments conducted in seventh and eighth grade science 
classrooms utilized different quiz and exam items as opposed to previously reported 
experiments where questions were the same with random assignment of multiple-choice 
items. Performance on quizzed material was significantly greater than non-quizzed material 
as the previous studies demonstrated. Significant testing effect was also demonstrated 
whether quizzes and exams were in the same or different formats. 

McDaniel et al. (2011) discovered that the testing effect was seen when students had items of 
an applied nature on a quiz and items of a conceptual nature on an exam, but not vice- 
versa. The testing effect stands to dispel the misconception that test and retest is a form of 
rote memorization. The test and retest cycle, which improves encoded information recall, 
shows that active processing of information is required for the testing effect phenomenon to 
occur (Yigit, Kiyici, & Cetinkaya, 2014). 

Roediger and Butler (2011b) have identified five points regarding the testing effect. First, 
testing produces better retention relative to passive studying methods (e.g., re-reading 
material). Second, testing repeatedly is more beneficial than taking a single test. Third, the 
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testing effect can be seen when no feedback (providing correct answers) is given after a test, 
however, providing feedback yields greater benefits. Fourth, sometime is required between 
being exposed to material and testing for retrieval practice to be beneficial. Finally, the 
benefits of the testing effect are not constrained to learning a specific response, rather, can be 
generalized to different contexts. 

A foil to the testing effect is the negative testing effect, where recall is poorer after subsequent 
restudy (Mulligan & Peterson, 2015). Tests consisting of free-recall items more consistently 
showed a negative testing effect in participants of Mulligan and Peterson's study. They 
believed the negative testing effect takes place due to the lack of mnemonic effect of episodic 
retrieval. Semantic retrieval of testing items were also not present in free-recall. The positive 
benefits of the testing effect are not present when test items are not actively encoded and 
given meaning (Jang, Pashler, & Huber, 2014). 

The testing effect is not without factors that can contribute to an increase in memory recall 
and decrease in time-dependent forgetting. Teachers are also factors in the testing effect 
phenomenon. The ability to encode information in a mnemonic and semantic manner can also 
be catalyzed by teachers who move from testing for testing's sake, toward a focus on the 
importance of what is being tested (Watson, Johanson, Loder, & Dankiw, 2014). Essentially, 
testing is a valuable tool that teachers can use to promote learning rather than merely 
evaluating what has been learned. 

Teachers are not the only part of the equation in developing a mastery-level approach to 
learning; student motivation is also a strong component. Self-regulated learning is a process 
that involves a learner transforming their mental abilities into academic skills (Zimmerman, 
2002). It refers to the behaviors and thought processes that learners engage in to attain goals 
they set for their learning. Hargis (2000) argues that students who are able to regulate their 
own learning with structure and discipline or have appropriate support ultimately perform 
better. Students that are better able to self-regulate have higher chances of learning, 
especially in an online environment. 

One such tool that supports the testing effect is an Internet-based application called 
Kahoot™. Kahoot™ is an online application where quizzes can be developed and presented in 
a "game-show" type format. Points are awarded for correct answers and participating 
students will immediately see the results of their responses. Game-based learning has the 
potential to be an effective tool for learning because it stimulates the visual and verbal 
components of our processing. (Woo, 2014). 

Laski and Siegler (2014) found the active encoding within game-based learning platforms to 
be effective in student learning by utilizing different styles of game in their study. A passive 
style of game, which did not provide meaning was less effective than game styles that utilized 
mental preparation and consideration. Plass, et al. (2013) found that although competitive 
nature of game-style learning activities did not have a noticeable trend in outcome, it did have 
a positive effect for a mastery goal orientation for learning. A study conducted on 58 eighth- 
grade students showed that there was an increase in mastery goal orientation toward learning 
when the game featured competition and collaboration (Plass, et. al., 2013). 

The purpose of this pilot quasi-experimental action research study is to introduce and assess 
the effectiveness of an alternate pedagogical approach for high-stakes test preparation that 
utilizes gamification through the use of Kahoot™. The basis of this pilot study is to see if the 
testing effect applies to students in higher education and if its use results in improved multiple- 
choice exam scores when compared to a control group. Confirming the efficacy of the testing 
effect by using a fun and engaging online application like Kahoot™ adds an empirically 
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supported pedagogical tool to the repertoire of university professors that has the potential to 
improve academic performance on high-stakes summative exams in a fun and engaging 
way. This pilot study will also add to a very limited amount of literature on the testing effect 
and gamification in a higher education setting. 

METHOD 

In order to address the challenge of low student scores on a high-stakes examination, two 
research questions were developed for this pilot action research study. The first question 
asked, 

> What is the difference in exam scores between students receiving course content 
through lecture, group discussions, and Kahoot™, and students who receive course 
content through only lecture and group discussions? 

The second question asked, 

> What kind of change can be brought about by engaging introductory psychology 
students with Kahoot™, an online quizzing application? 

RESEARCH DESIGN 

This pilot quasi-experimental action research study occurred in this researcher's General 
Psychology class, which consisted of two separate sections. Prior to the start of the semester, 
the researcher flipped a coin to determine which section would be the experimental group and 
which would be the control group. The online application Kahoot™ was introduced to the 
students in the experimental group. 

Both groups received the same syllabi, lectures, viewed the same videos and presentation 
slides, and conversed about the same topics during in-class group discussions. The difference 
between the two groups occurred in the last 10 minutes of pre-determined classes. For the 
experimental group, the researcher would close up the discussion approximately 10 minutes 
prior to the end of class. The control group would continue on through the end of class. 

During those final 10 minutes, the experimental group participated in the online no-stakes 
quizzes (7 multiple-choice questions per chapter) through the use of Kahoot™. Questions for 
Kahoot™ were written by the researcher based on the key concepts that were going to be on 
the upcoming high-stakes exam. Questions in Kahoot™ aligned with the key concepts, but the 
actual wording of the questions were not identical to the questions used for the high-stakes 
exam. Questions for the high-stakes exam were chosen from a test bank that was associated 
with the assigned text for the course. 

Students who chose to participate by logging into Kahoot™ would create a screen name (using 
their real name was discouraged to encourage anonymity) using any Internet browser on any 
type of mobile device (e.g., laptop, smart phone, tablet). As questions appeared on the screen, 
students would select their response on their mobile device. Once all of the students 
responded or the preset time (20 seconds) elapsed, students received immediate feedback on 
the correct answer. The top five screen names who responded accurately and the quickest 
would appear on the screen for all to see. Students were made aware that the results of these 
online quizzes had no impact on their course grade. 

Both the control and experimental groups took the same high-stakes multiple-choice exam in 
their respective classes on the same day. In preparation, the control group received a study 
guide with all of the key concepts that were also covered in the Kahoot™ questions. In 
addition, an entire class period (50 minutes) was spent reviewing all of the concepts in a 
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lecture-based format. Students in the control group were allowed to ask questions any time 
during that class period. In addition, the researcher was available during office hours for 
students who wanted to discuss the material in more depth. The experimental group did not 
receive a study guide. Instead, the experimental group spent an entire class period (50 
minutes) replaying all of the Kahoot™ quizzes. After all of the quizzes were completed, 
students were allowed to ask questions. In addition, the researcher was available during office 
hours for students who wanted to discuss the material in more depth. Both groups received 
the key concepts that were going to be tested on the high-stakes exam, but the control group 
received it in writing and through a lecture versus the experimental group who received it in 
the form of no-stakes quizzes using Kahoot™. 

The mean score of both groups were analyzed to determine if a statistically significant difference 
between the two data sets existed. In addition, the researcher gathered qualitative feedback from 
all participants through the use of a questionnaire. Qualitative responses were thematically 
analyzed to determine trends and also to receive feedback about their preparation experience 
leading up to the high-stakes exam. 

Recruitment Strategy 

As the primary stakeholder for the findings from this study, this researcher's positionality as a 
scholar-practitioner is a key element of this quasi-experimental action research study (Herr & 
Anderson, 2005). Since the purpose of this action research study was to determine an alternate 
pedagogical approach to increase student performance in the researcher's class, utilizing a 
convenience sample that consisted of the researcher's students was found to be appropriate. 
Although a convenience sample is commonly defined as a sample that is easiest to access, having 
the researcher's students be involved appropriately addressed this study's two research questions 
by providing useful qualitative and quantitative data directly from the study's target population. 

Post-secondary students (18 years or older) from this researcher's undergraduate General 
Psychology course were sampled for this pilot quasi-experimental action research study. The only 
exclusion criteria is that participants cannot be considered a minor (under the age of 18). Students 
who decide to opt out of the study will not receive any type of penalty or loss of points. Their exam 
score (dependent variable) will not be included in the calculation of the mean score. 

Sample 

The sample size was 49 undergraduate psychology students enrolled in a first-year general 
psychology course. The student population at this research site consists of 68% females and 
32% males. Sixty-seven percent is Asian/Pacific Islander, 17% White, non-Hispanic, 6% 
Hispanic, 4% African-American, and 6% other. The diversity within the sample groups was 
representative of this. 

Data Analysis 

The exam is multiple-choice utilizing Scantron software for grading. The mean exam scores 
will be statistically analyzed using an Independent-Samples T-Test. An Independent-Samples 
T-Test tests for statistical significance when comparing the mean scores of two groups 
consisting of interval and ratio level data. 

The researcher gathered qualitative feedback through the use of the following questionnaire: 

1) What helped you in class prepare for the exam? 

2) During class, what helped you comprehend the material being presented? 

3) What would have supported your learning more if implemented? 

For the experimental group only: 

4) What effect did the Kahoot™ quizzes have on your preparation for the exam? 
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Qualitative responses were thematically analyzed to determine learning trends and for 
feedback about the efficacy of the preparation methods implemented by the researcher. In 
addition, responses were analyzed for common words and phrases using a text analyzer 
developed by online-utility.org (The text analyzer can be accessed through 
http://www.online-utility.org/text/analyzer.jsp). This questionnaire is a form of Classroom 
Assessment Technique (CAT) created by the primary author. 

Confidentiality 
Exam scores 

> Administered using Scantron; 

> Participants' results data were entered into SPSS (Statistical Package for the Social 
Sciences) by the researcher to administer T-Test, which compares mean scores of the 
experimental and control group; 

> Electronic data were password protected on this researcher's office computer; 

> All paper copies of student scores were protected with a double lock system in this 
researcher's private university office; and 

> All of the examination data were kept securely for one year after the completion of this 
study. 

Questionnaire 

> Upon completing the questionnaire a thematic analysis was conducted to assess the 
students' perceptions and practices in preparing for the exam; 

> After the analysis has been conducted by this researcher, all of the questionnaires were 
protected with a double lock system in this researcher's private university office; and 

> All of the questionnaires are kept securely for one year after the completion of this 
study. 

Informed Consent 

This researcher introduced the study in each class. Students were able to ask questions. The 
researcher provided each student a link to a Google Form, which included the informed consent 
form as the first page the student would see. Students needed to agree with the informed 
consent form prior to gaining access to the survey. No participants were minors. The 
researcher emphasized when introducing this study to each class that no penalty of any kind 
will be received if a student wishes to opt out of this study. 

Potential Risks to Participants 

The pilot research study is not more than minimal risk. Research occurred in an established 
educational institution and setting, which involved normal educational practices. 

Potential Benefits to Participants 

Participants learned an alternate studying tool to assist them in taking future high-stakes 
exams. In addition, the literature on the testing effect consistently shows that practicing 
taking tests enhances a person's memory retrieval skills when taking a high-stakes 
exam. Kahoot™ is used primarily in the K-12 setting and after an exhaustive literature review 
no studies involving Kahoot™ were conducted in a higher education setting. Because Kahoot™ 
engages students through game play, the experience of taking frequent low to no-stakes test 
should not be as threatening or intimidating when compared to frequent low to no-stakes tests 
administered traditionally through paper and pen/pencil. The primary benefit is adding an 
empirically proven tool that university professors can implement that will have a positive 
influence on academic performance in high-stakes exams. 
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FINDINGS 


The source of the data was derived from the following: 

(a) multiple-choice examinations, and 

(b) a student dispositional survey. 

An Independent-Samples T-Test was conducted to compare the effects of the online 
application Kahoot™ on the mean test scores in the experimental and control 
groups. Significance was determined between the two groups at the p<.05 level (F(l, 47) = 
7.801, p = .008). These results suggest that the use of Kahoot did have a significant effect on 
test scores beyond chance. 

Through the utilization of the online text analyzer the following was also determined. In 
response to the first question, "What helped you in class prepare for the exam," participants 
in the control group (n=23) mentioned "PowerPoint" 11 times followed by "notes" (9) and 
"study guide" (8). Participants in the experimental group (n=24) mentioned "Kahoot™" 17 
times (see Tables 1 and 2). 

Tables 3 and 4 indicate results of the 2nd survey question, "During class, what helped you 
comprehend the material?" Participants in both the control group (n=22) and experimental 
group (n=24) mentioned "videos" (14 and 7 respectively), however the experimental group 
responded with "Kahoot™" 10 times. 

The third survey question that was asked to both control and experimental groups was, "What 
would have supported your learning more if implemented." The control group (n=23) 
mentioned both "quizzes" and "activities" 3 times. The experimental group (n=24) responded 
with "discussions" 4 times (see tables 5 and 6). 

The final question was asked only to the experimental group, "What effect did the Kahoot 
quizzes have on your preparation for the exam?" Table 7 shows that participants responded 
with "helped" 17 times, followed by "fun" (3), "good" (3), and "positive" (2). 


Table: 1 

Control Group: What helped you in class prepare for the exam? (n = 23 responses) 


Word Used 

Times Mentioned 

PowerPoint 

11 

Notes 

9 

Study Guide 

8 

Videos 

6 

Lectures 

3 

Discussion 

2 

Labs 

2 

Textbooks 

2 
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Table: 2 

Experimental Group: What helped you in class prepare for the exam? (n = 24 responses) 


Word Used 



Times Mentioned 

Kahoot 



17 

Notes 



5 

Review 



4 

Quizzes 



3 

PowerPoints 



2 

Table: 3 

Control Group: During class, what helped you comprehend the material being presented? 

(n = 22 responses) 

Word Used 



Times Mentioned 

Videos 



14 

PowerPoints 



7 

Group 



3 

Labs 



3 

Discussions 



2 

Lecture 



2 

Relate to Real Life Situations 


2 


Table: 4 

Experimental Group: During class, what helped you comprehend the material 
_ being presented? (n - 24 responses) _ 


Word Used 

Times Mentioned 

Kahoot 


10 

Videos 


7 

PowerPoints 


3 

Notes 


2 

Discussions 


2 
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Table: 5 

Control Group: What would have supported your learning more if implemented? 

(n = 23 responses) 


Word Used 


Times Mentioned 

Quizzes 


3 

More Activities 


3 

Table: 6 

Experimental Group: What would have supported your learning more if implemented? 

(n = 24 responses) 

Word Used 


Times Mentioned 

Discussions 


4 


Table: 7 

Experimental Group: What effect did the Kahoot quizzes have on your preparation 

for the exam? (n = 24 responses) 


Word Used 

Times Mentioned 

Helped 


17 

Fun 


3 

Good 


3 

Positive 


2 


DISCUSSION and CONCLUSION 

Upon reflection of this study's findings, it became apparent that the common theme discovered 
from both the experimental and control group was the importance of applied 
learning. Participants consistently expressed the need for active reviewing, visual learning, 
and making the content meaningful to them by applying what they learned to a form of 
experience. Although the means of achieving this theme varied, the theme of applied learning 
was consistent in the analysis of the feedback provided by both groups of participants. The 
more active the learning process the more effective that method appeared to be. This was 
evident in the findings in response to the two research questions. 

The first research question asked "what is the difference in exam scores between students 
receiving course content through lecture, group discussions, and Kahoot™ and students who 
receive course content through only lecture and group discussions?" The statistical analysis 
comparing the examination scores of the experimental and control groups suggest that the 
testing effect, through the use of Kahoot™, had a significant impact on academic performance 
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of the experimental group when compared to the control group. The qualitative findings also 
supported this analysis by providing feedback to the researcher about the efficacy of a lecture- 
based versus applied learning form of preparation for high-stakes exams. In particular, 
students appreciated the immediate feedback from Kahoot™ as it provided them a real-time 
gauge of where they stood in the class. Students also felt that being exposed to questions 
leading up to the examination helped them feel more comfortable when taking the high-stakes 
examination. Students also mentioned looking forward to class because they enjoyed playing 
Kahoot™ as it helped them to memorize key concepts of the class. Kahoot™ was the only 
pedagogical tool used that had no criticism of any kind in the survey. Students in the control 
group, who did not have access to Kahoot™, listed visual versus auditory tools highest on their 
preferred list of preparation tools. This supports the growing scholarship of teaching and 
learning (SoTL) literature that finds lecture-based pedagogy as less effective then student- 
focused approaches. 

The second research question asked "what kind of change can be brought about by engaging 
introductory psychology students with Kahoot™, an online quizzing application?" The general 
consensus of the experimental group felt that Kahoot™ was an effective pedagogical tool 
because the students found it fun and engaging. This finding was impactful as students do not 
always welcome and actively engage pedagogical changes (Iwamoto, et al., 2016). Students 
commented on how they enjoyed the competitive aspect of Kahoot™. Students who preferred 
not to engage in the competitiveness could still participate as they could remain anonymous 
by using a fictitious screen name. Students in the experimental group were observed having 
fun and showing enthusiasm especially when seeing their screen names on the 
leaderboard. There were also comments made in class where students would prepare for class 
(e.g., read the text) because they knew that Kahoot™ would be played. This level of 
enthusiasm was not observed in the control group. The energy level, engagement, and even 
relevant class dialogue was observed to be higher in the experimental group. An interesting 
side effect to this was an increase use of peer study groups. Students in the experimental 
group were observed working together and supporting each other more than in the control 
group. Because students in the class were having fun with one another, starting conversations 
with one another appeared to be easier and less intimidating. This was not observed in the 
control group. Comradery was not as evident. 

The utilization of games in the classroom setting has been shown to be an effective 
pedagogical tool to improve academic performance. It appears that this form of teaching and 
learning aligns with our culture's current demand for mobile applications and video 
games. Students can regularly be seen around campus using their mobile devices for 
communication, entertainment, and learning. The vast majority of students have a high 
degree of comfort using technology to learn. There was a very small learning curve when 
Kahoot™ was introduced to the experimental group. In fact, approximately one-fourth of the 
class had played Kahoot™ at some point during their K-12 schooling. 

It can be suggested that pedagogical tools like Kahoot™ have the potential to enhance and 
improve high-stakes examination scores at the college and university level. Students in the 
experimental group felt positive about their experience. The results of this study also suggest 
that creating a fun and engaging environment also supports improved academic 
performance. Students will learn what excites them. If a student cares about what she or he 
is introduced to, she or he will be motivated to learn. 
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LIMITATIONS 


This pilot quasi-experimental action research study was limited by its relatively small sample 
size (/?= 49) and convenience sample. Although it began to answer the research questions for 
the researcher as it applied to the researcher's class, utilizing a larger sample size at various 
levels (first-year, second-year, third-year, and fourth-years and higher) and from additional 
institutes of higher education would substantially increase the transferability of this 
study. Also, the fact that there was familiarity with Kahoot™ prior to this study could be 
considered a confounding variable. Another limitation was that the students' self-regulation 
skills were not assessed. By assessing self-regulation skills, the students' study skills could be 
analyzed to determine if that was another confounding variable in the students' performance 
with the high-stakes exam. 

RECOMMENDATIONS FOR FUTURE RESEARCH 

Recommendations for further research emerging from this action research study include; 

> Future action research cycles should collect and analyze data over a longer period of 
time (e.g., the entire semester) versus just the first half of the semester. 

> A study looking into the effectiveness of gamification at the higher education level 
would clarify if the significant findings from this study was due specifically to the 
online application Kahoot™ or the experience of learning through the gamification 
of the course's content. 

> A study that looks into the self-regulation skills of students to determine how much 
time and energy they are putting into their examination preparation and if there is 
a positive correlation between that and academic performance. 

> Develop a baseline assessment to determine the level of variance in the comparison 
groups. This will increase generalizability and transferability of the findings. 
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